1. Field of the Invention
The present invention relates to an image processing method and apparatus, and specifically to low-light video frame enhancement.
2. Description of the Related Art
Sensor arrays for digital cameras exist for capturing color photos. Sensors known as RGBW sensors are used for capturing red, green, and blue colors, and for capturing luminance information for multiple pixels of an array of pixels. The red, green and blue pixels include filters such that only certain narrow ranges of wavelengths of incident light are counted. The white pixels capture light of red, green and blue wavelengths, i.e., of a broader range of wavelengths than any of the blue, green and red pixels. Thus, the white pixels are typically brighter than any of the blue, red and green pixel if they are exposed for the same duration. Noise removal algorithms can tend to blur face regions in an undesirable manner. Noise removal algorithms are described at U.S. patent application Ser. Nos. 11/856,721 and 11/861,257, which are hereby incorporated by reference, as are U.S. Ser. Nos. 10/985,650, 11/573,713, 11/421,027, 11/673,560, 11/319,766, 11/744,020, 11/753,098, 11/752,925, and 12/137,113, which are assigned to the same assignee as the present application and are hereby incorporated by reference.
Kodak has developed a RGBW color filter pattern differing from the previously known Bayer Color Filter. The RGBW pattern of Kodak is referred to as a Color Filter Array (CFA) 2.0. One half of cells in a RGBW pattern are panchromatic, i.e. sensing all of the color spectrum (Y component)—usually called white cells. This way more light energy is accumulated in the same amount of time than for color pixels. A Bayer filter uses only ⅓ (˜0.33) of color spectrum energy. An RGBW filter uses 4/6 (˜0.67) of the energy, where ½ comes from white cells and ⅙ from RGB cells.
CFA Array looks something like the following:                WBWG . . .        BWGW . . .        WGWR . . .        RWRW . . .        
In this context, the following are incorporated by reference: U.S. Pat. Nos. 7,195,848, 7,180,238, 7,160,573, 7,019,331, 6,863,368, 6,607,873, 6,602,656, 6,599,668, 6,555,278, 6,387,577, 6,365,304, 6,330,029, 6,326,108, 6,297,071, 6,114,075, 5,981,112, 5,889,554, 5,889,277, 5,756,240, 5,756,239, 5,747,199, 5,686,383, 5,599,766, 5,510,215, 5,374,956, and 5,251,019.
Two source images nominally of the same scene may be used to produce a single target image of better quality or higher resolution than either of the source images.
In super-resolution, multiple differently exposed lower resolution images can be combined to produce a single high resolution image of a scene, for example, as disclosed in “High-Resolution Image Reconstruction from Multiple Differently Exposed Images”, Gunturk et al., IEEE Signal Processing Letters, Vol. 13, No. 4, April 2006; or “Optimizing and Learning for Super-resolution”, Lyndsey Pickup et al, BMVC 2006, 4-7 Sep. 2006, Edinburgh, UK, which are hereby incorporated by reference. However, in super-resolution, blurring of the individual source images either because of camera or subject motion are usually not of concern before the combination of the source images.
U.S. Pat. No. 7,072,525, incorporated by reference, discloses adaptive filtering of a target version of an image that has been produced by processing an original version of the image to mitigate the effects of processing including adaptive gain noise, up-sampling artifacts or compression artifacts.
US published applications 2006/0098890, 2007/0058073, 2006/0098237, 2006/0098891, European patent EP1779322B1, and PCT Application No. PCT/EP2005/011011 (WO2006/050782), which are each hereby incorporated by reference, describe uses of information from one or more presumed-sharp, short exposure time (SET) preview images to calculate a motion function for a fully exposed higher resolution main image to assist in the de-blurring of the main image.
Indeed many other documents, including US 2006/0187308, Suk Hwan Lim et al.; and “Image Deblurring with Blurred/Noisy Image Pairs”, Lu Yuan et al, SIGGRAPH07, Aug. 5-9, 2007, San Diego, Calif., which are incorporated by reference, are directed towards attempting to calculate a blur function in the main image using a second reference image before de-blurring the main image.
Other approaches, such as may be disclosed in US2006/0017837, which is incorporated by reference, involve selecting information from two or more images, having varying exposure times, to reconstruct a target image where image information is selected from zones with high image details in SET images and from zones with low image details in longer exposure time images.
Nowadays, even though image processing techniques have evolved, still low-light scenes tend to lead to very dark frames in videos. By these means the visual quality is low. For example, movies taken in low-light conditions (e.g. 30 lux illumination), at 30 fps, generally result in very dark, underexposed frames that have a very low contrast, so objects are not distinguishable. Tone mapping (amplification) algorithms may be applied to improve such images. Although these may provide a better exposed image, there are difficulties in restoring the natural colors of the scene. For example, simple summation or forcing higher exposure tends to lead to frames degraded by motion blur.
United States published patent application no. 2009/0303343, which belongs to the same assignee and is incorporated by reference, describes background techniques generally involving two different images acquired either at different moments of time or by two different sensors. It is desired to have a new technique that utilizes images having identical or substantially similar content, and such is provided herein.